1. Technical Field
The present invention relates generally to analyzing and processing hierarchical data structures represented as directed graphs, and more particularly relates to a system and method for providing an optimized parallel analysis of hierarchical data. This includes directed graph representations of tree's and other non-cyclic graphs.
2. Related Art
The collection, processing, and analysis of electronic information have become critical tools in today's business methodologies. For instance, businesses recognize the importance of leveraging electronic information in order to more efficiently deliver goods and services to consumers. Analysis of such information might provide important business intelligence, reveal consumer interests, and determine marketplace trends.
However, as the amount of electronic information that is generated continues to grow, the ability to effectively and efficiently analyze the information becomes more challenging. Specifically, there is a significant time and cost factor involved in analyzing and processing large databases of information (i.e., data mining). For many large institutions, it may take on the order of hours to analyze or “walk through” a database of electronic information. The problem is further exacerbated when different processing or analysis tools must be run on the same set of data. In this case, the time and cost factor is multiplied by the number of required tools.
One method for handling large databases is to store, or arrange the information in a hierarchy. Directed non-cyclic graphs comprise nodes arranged in a generally hierarchical fashion, with the more general information stored at the top, and more specific information branched off below, terminating in leaf nodes. Leaf nodes do not have children (directed lines from the leaf node to another node). In some cases, information stored at the lower levels of the hierarchy may be referenced by multiple nodes higher in the graph, representing multiple references to common, or shared, information. Since the graph is non-cyclic, there is never a path from any given node that arrives back at that same node.
U.S. Pat. No. 6,055,539, entitled METHOD TO REDUCE I/O FOR HIERARCHICAL DATA PARTITIONING METHODS, issued on Apr. 25, 2000, which is hereby incorporated by reference, describes a method for classifying hierarchical data to allow for more efficient processing of a data set.
While data hierarchies and related methods for storing data have improved the process of analyzing large databases, the prior art has failed to adequately solve the problem of efficiently performing multiple processes or applying multiple tools to a set of hierarchical data. For example, a straightforward recursive descent walk of a hierarchical data tree (a tree is a simple directed non-cyclic graph) of N nodes by M analysis routines would require N*M visits to the individual nodes. As is evident, a more efficient mechanism for handling this situation is required.
“Design Patterns—Elements of Reusable Object-Oriented Software,” by Gamma, Helm, Johnson, and Vlissides (ISBN 0-201-63361-2), pages 293-303, copyright 1995, which is hereby incorporated by reference, describes an ‘Observer’, or ‘Publish-Subscribe’ design pattern. While not directly applicable to a static directed graph, the mechanism describes an ‘event drive’ model for processing data that has been used to allow multiple analysis routines access to the hierarchical data. Such algorithms are frequently used in the processing of XML, for example the Simple API for XML (SAX) uses this event driven model. However, this disclosure fails to provide efficient optimization techniques for handling such data.